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Abstract 

We define the "Pulse Synchronization" problem that requires nodes to 
achieve tight synchronization of regular pulse events, in the settings of dis- 
tributed computing systems. Pulse-coupled synchronization is a phenomenon 
displayed by a large variety of biological systems, typically overcoming a 
high level of noise. Inspired by such biological models, a robust and self- 
stabilizing Byzantine pulse synchronization algorithm for distributed com- 
puter systems is presented. The algorithm attains near optimal synchroniza- 
tion tightness while tolerating up to a third of the nodes exhibiting Byzantine 
behavior concurrently. Pulse synchronization has been previously shown to 
be a powerful building block for designing algorithms in this severe fault 
model. We have previously shown how to stabilize general Byzantine algo- 
rithms, using pulse synchronization. To the best of our knowledge there is no 
other scheme to do this without the use of synchronized pulses. 

Keywords: Self-stabilization, Byzantine faults, Distributed algorithms, Robust- 
ness, Pulse synchronization, Biological synchronization, Biological oscillators. 



1 Introduction 

The phenomenon of synchronization is displayed by many biological systems [32]. 
It presumably plays an important role in these systems. For example, the heart of 
the lobster is regularly activated by the synchronized firing of four interneurons in 
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the cardiac pacemaker network [16, 17]. It was concluded that the organism cannot 
survive if all four interneurons fire out of synchrony for prolonged times [30]. This 
system inspired the present work. Other examples of biological synchronization 
include the malaccae fireflies in Southeast Asia where thousands of male fireflies 
congregate in mangrove trees, flashing in synchrony [4]; oscillations of the neurons 
in the circadian pacemaker, determining the day-night rhythm; crickets that chirp 
in unison [33]; coordinated mass spawning in corals and even audience clapping 
together after a "good" performance [28]. Synchronization in these systems is 
typically attained despite the inherent variations among the participating elements, 
or the presence of noise from external sources or from participating elements. A 
generic mathematical model for synchronous firing of biological oscillators based 
on a model of the human cardiac pacemaker is given in [27]. This model does not 
account for noise or for the inherent differences among biological elements. 

In computer science, synchronization is both a goal by itself and a building 
block for algorithms that solve other problems. In the "Clock Synchronization" 
problem, it is required of computers to have their clocks set as close as possible to 
each other as well as to keep a notion of real-time ([11, 21, 22]). 

In general, it is desired for algorithms to guarantee correct behavior of the sys- 
tem in face of faults or failing elements, without strong assumptions on the initial 
state of the system. It has been suggested in [30] that similar fault considerations 
may have been involved in the evolution of distributed biological systems. In the 
example of the cardiac pacemaker network of the lobster, it was concluded that at 
least four neurons are needed in order to overcome the presence of one faulty neu- 
ron, though supposedly one neuron suffices to activate the heart. The cardiac pace- 
maker network must be able to adjust the pace of the synchronized firing according 
to the required heartbeat, up to a certain bound, without losing the synchrony (e.g. 
while escaping a predator a higher heartbeat is required - though not too high). 
Due to the vitality of this network, it is presumably optimized for fault tolerance, 
self-stabilization, tight synchronization and for fast re-synchronization. 

The apparent resemblance of the synchronization and fault tolerance require- 
ments of biological networks and distributed computer networks makes it very ap- 
pealing to infer from models of biological systems onto the design of distributed 
algorithms in computer science. Especially when assuming that distributed biolog- 
ical networks have evolved over time to particularly tolerate inherent heterogeneity 
of the cells, noise and cell death. In the current paper, we show that in spite of obvi- 
ous differences, a biological fault tolerant synchronization model ([30]) can inspire 
a novel solution to an apparently similar problem in computer science. 

We propose a relaxed version of the Clock Synchronization problem, which 
we call "Pulse Synchronization", in which all the elements are required to invoke 
some regular pulse (or perform a "task") in tight synchrony, but allows to deviate 
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from exact regularity. Though nodes need to invoke the pulses synchronously, 
there is a limit on how frequently it is allowed to be invoked (similar to the linear 
envelope clock synchronization limitation). The "Pulse Synchronization" problem 
resembles physical/biological pulse-coupled synchronization models [27], though 
in a computer system setting an algorithm needs to be supplied for the nodes to 
reach the synchronization requirement. To the best of our knowledge this problem 
has not been formally defined in the settings of distributed computer systems. 

We present a novel algorithm in the settings of self-stabilizing distributed algo- 
rithms, instructing the nodes how and when to invoke a pulse in order to meet the 
synchronization requirements of "Pulse Synchronization". The core elements of 
the algorithm are analogous to the neurobiological principles of endogenous (self 
generated) periodic spiking, summation and time dependent refractoriness. The 
basic algorithm is quite simple: every node invokes a pulse regularly and sends a 
message upon invoking it {endogenous periodic spiking). The node sums messages 
received in some "window of time" {summation) and compares this to the con- 
tinuously decreasing time dependent firing threshold for invoking the pulse {time 
dependent refractory function). The node fires when the counter of the summed 
messages crosses the current threshold level, and then resets its cycle. For in-depth 
explanations of these neurobiological terms see [20]. 

The algorithm performs correctly as long as less than a third of the nodes be- 
have in a completely arbitrary ("Byzantine") manner concurrently. It ensures a 
tight synchronization of the pulses of all correct nodes, while not using any central 
clock or global pulse. We assume the communication network allows for a broad- 
cast environment and has a bounded delay on message transmission. The algorithm 
may not reach its goal as long as these limitations are violated or the network graph 
is disconnected. The algorithm is self-stabilizing Byzantine and thus copes with a 
more severe fault model than the traditional Byzantine fault model. Classic Byzan- 
tine algorithms, which are not designed with self-stabilization in mind, typically 
make use of assumptions on the initial state of the system such as assuming all 
clocks are initially synchronized, (c.f. [11]). Observe that the system might tem- 
porarily be thrown out of the assumption boundaries, e.g. when more than one 
third of the nodes are Byzantine or messages of correct nodes get lost. When the 
system eventually returns to behave according to these presumed assumptions it 
may be in an arbitrary state. A classic Byzantine algorithm, being non-stabilizing, 
might not recover from this state. On the other hand, a self-stabilizing protocol 
converges to its goal from any state once the system behaves well again, but is 
typically not resilient to permanent faults. For our protocol, once the system com- 
plies with the theoretically required bound of / < 3n permanent Byzantine faulty 
nodes in a network of n nodes then, regardless of the state of the system, tight pulse 
synchronization is achieved within finite time. It overcomes transient failures and 
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permanent Byzantine faults and makes no assumptions on any initial synchronized 
activity among the nodes (such as having a common reference to time or a common 
event for triggering initialization). 

Our algorithm is uniform, all nodes execute an identical algorithm. It does not 
suffer from communication deadlock, as can happen in message-driven algorithms 
([3]), since the nodes have a time-dependent state change, at the end of which they 
fire endogenously. The faulty nodes cannot ruin an already attained synchroniza- 
tion; in the worst case, they can slow down the convergence towards synchroniza- 
tion and speed up the synchronized firing frequency up to a certain bound. The 
convergence time is O(f) cycles with a near optimal synchronization of the pulses 
to within d real-time (the bound on the end to end network and processing delay). 
We show in Subsection 3.3 how the algorithm can be executed in a non-broadcast 
network to achieve synchronization of the pulses to within 3d real-time. 

Applications and contribution of this paper: We have shown in [6] how to 
stabilize general Byzantine algorithms using synchronized pulses. In [8] we have 
presented a very efficient, besides being the first, self-stabilizing Byzantine token 
passing algorithm. The efficient self-stabilizing Byzantine clock synchronization 
algorithm in [5] is also the first such algorithm for clock synchronization. All these 
algorithms assume a background self-stabilizing Byzantine pulse synchronization 
module though the particular pulse synchronization procedure presented in [5] suf- 
fers from a flaw 1 . The only other self-stabilizing Byzantine pulse synchronization 
algorithm (besides the current work), is to the best of our knowledge, the one in 
[9], which is a correction to the one in [5]. In comparison to the current paper, the 
pulse synchronization algorithm in [9] has a much higher message complexity and 
worse tightness, is more complicated but it converges in 0(1), does not assume 
broadcast and scales better. The current paper is simpler, uses much shorter mes- 
sages; it has a smaller message complexity and introduces novel and interesting 
elements to distributed computing. 

In the Discussion, in Section 6, we postulate that our result elucidates the fea- 
sibility and adds a solid brick to the motivation to search for and to understand 
biological mechanisms for robustness that can be carried over to computer sys- 
tems. 

2 Model and Problem Definition 

The environment is a network of n nodes, out of which / are faulty nodes, that com- 
municate by exchanging messages. The nodes regularly invoke "pulses", ideally 

'The flaw was pointed out by Mahyar Malekpour from NASA LaRC and Radu Siminiceanu from 
NIA, see [25] . 
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every Cycle real-time units. The invocation of the pulse is expressed by sending 
a message to all the nodes; this is also referred to as firing. We assume that the 
message passing allows for an authenticated identity of the senders. The commu- 
nication network does not guarantee any order on messages among different nodes. 
Individual nodes have no access to a central clock and there is no external pulse 
system. The hardware clock rate (referred to as the physical timers) of correct 
nodes has a bounded drift, p, from real-time rate. When the system is not coher- 
ent then there can be an unbounded number of concurrent Byzantine faulty nodes, 
the turnover rate between faulty and non-faulty nodes can be arbitrarily large and 
the communication network may behave arbitrarily. Eventually the system settles 
down in a coherent state in which there at most / < 3n permanent Byzantine faulty 
nodes and the communication network delivers messages within bounded time. 

Definition 2.1. A node is non-faulty at times that it complies with the following: 

1. (Bounded Drift) Obeys a global constant < p « 1 (typically p 10 -6 ), 
such that for every real-time interval [u, v] : 

(1 — p)(v — u) < 'physical timer' (v)— 'physical timer' (u) < (l+p)(v — u). 

2. (Obedience) Operates according to the correct protocol. 

3. (Bounded Processing Time) Processes any message of the correct protocol 
within 7r real-time units of arrival time. 

A node is considered faulty if it violates any of the above conditions. The 
faulty nodes can be Byzantine. A faulty node may recover from its faulty behavior 
once it resumes obeying the conditions of a non-faulty node. In order to keep the 
definitions consistent the "correction" is not immediate but rather takes a certain 
amount of time during which the non-faulty node is still not counted as a correct 
node, although it supposedly behaves "correctly" 2 . We later specify the time-length 
of continuous non-faulty behavior required of a recovering node to be considered 
correct. 

Definition 2.2. The communication network is non-faulty at periods that it com- 
plies with the following: 

• (Bounded Transmission Delay) Any message sent or received by a non-faulty 
node will arrive at every non-faulty node within 5 real-time units. 

2 For example, a node may recover with arbitrary variables, which may violate the validity condi- 
tion if considered correct immediately. 
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Thus, our communication network model is an "eventual bounded-delay" com- 
munication network. 

Basic definitions and notations: 

We use the following notations though nodes do not need to maintain all of 
them as variables. 

• d = 5 + 7r. Thus, when the communication network is non-faulty, d is the 
upper bound on the elapsed real-time from the sending of a message by a 
non-faulty node until it is received and processed by every correct node. 

• A pulse is an internal event targeted to happen in "tight" 3 synchrony at all 
correct nodes. A Cycle is the "ideal" time interval length between two suc- 
cessive pulses that a node invokes, as given by the user. The actual cycle 
length, denoted in regular caption, has upper and lower bounds as a result of 
faulty nodes and the physical clock skew. 

• a represents the upper bound on the real-time window within which all cor- 
rect nodes invoke a pulse {tightness of pulse synchronization). Our solution 
achieves a = d. We assume that Cycle S> a. 

• (f>i{t) € R + U {oo}, < i < n, denotes, at real-time t, the elapsed real-time 
since the last pulse invocation of pi. It is also denoted as the "4> of node p". 
We occasionally omit the reference to the time in case it is clear out of the 
context. For a node, pj , that has not fired since initialization of the system, 

(f)j = oo. 

• cycle m j n and cycle max are values that define the bounds on the actual cycle 
length during correct behavior. We achieve 

n — 2/ 

cycle min = — — — • Cycle ■ (1 - p) < cycle < Cycle • (1 + p) = cycle max . 
n j 

• message_decay represents the maximal real-time a non-faulty node will keep 
a message or a reference to it, before deleting it 4 . 

In accordance with Definition 2.2, the network model in this paper is such that 
every message sent or received by a non-faulty node arrives within bounded time, 
8, at all non-faulty nodes. The algorithm and its respective proofs are specified in 
a stronger network model in which every message received by a non-faulty node 

3 We consider c ■ d, for some small constant c, as tight. 

4 The exact elapsed time until deleting a messages is specified in the PRUNE procedure in Fig. 2. 
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arrives within S time at all non-faulty nodes. The subtle difference in the latter 
definition equals the assumption that every message received by a non-faulty node, 
even a message from a Byzantine node, will eventually reach every non-faulty 
node. This weakens the possibility for two-faced behavior by Byzantine nodes. 
The algorithm is able to utilize this fact so that if executed in such a network en- 
vironment, then it can attain a very tight, near optimal, pulse synchronization of 
d real-time units. We show in Subsection 3.3 how to execute in the background a 
self-stabilizing Byzantine reliable-broadcast-like primitive, which executes in the 
network model of Definition 2.2. This primitive effectively relays every message 
received by a non-faulty node so that the latter network model is satisfied. In such 
a case the algorithm can be executed in the network model of Definition 2.2 and 
achieves synchronization of the pulses to within 3d real-time. 

Note that the protocol parameters n, f and Cycle (as well as the system char- 
acteristics d and p) are fixed constants and thus considered part of the incorruptible 
correct code 5 . Thus we assume that non-faulty nodes do not hold arbitrary values 
of these constants. 

A recovering node should be considered correct only once it has been continu- 
ously non-faulty for enough time to enable it to have decayed old messages and to 
have exchanged information with the other nodes through at least a cycle. 

Definition 2.3. A node is correct following cycle max + a + message_decay real- 
time of continuous non-faulty behavior. 

Definition 2.4. The communication network is correct following cycle max + a + 
message_decay real-time of continuous non-faulty behavior. 

Definition 2.5. (System Coherence) The system is said to be coherent at times 
that it complies with the following: 

1. (Quorum) There are at least n — f correct nodes, where f is the upper bound 
on the number of potentially non-correct nodes, at steady state. 

2. (Network Correctness) The communication network is correct. 

Hence, if the system is not coherent then there can be an unbounded number of 
concurrent faulty nodes; the turnover rate between the faulty and non-faulty nodes 
can be arbitrarily large and the communication network may deliver messages with 
unbounded delays, if at all. The system is considered coherent, once the commu- 
nication network and a sufficient fraction of the nodes have been non-faulty for a 

5 A system cannot self-stabilize if the entire code space can be perturbed, see [15]. 
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sufficiently long time period for the pre-conditions for convergence of the proto- 
col to hold. The assumption in this paper, as underlies any other self-stabilizing 
algorithm, is that the system eventually becomes coherent. 

All the lemmata, theorems, corollaries and definitions hold as long as the sys- 
tem is coherent. 

We now seek to give an accurate and formal definition of the notion of pulse 
synchronization. We start by defining a subset of the system states, which we call 
pulse_states, that are determined only by the elapsed real-time since each individ- 
ual node invoked a pulse (the 0's). We then identify a subset of the pulse_states in 
which some set of correct nodes have "tight" or "close" 0's. We refer to such a set 
as a synchronized set of nodes. To complete the definition of synchrony there is a 
need to address the recurring brief time period in which a correct node in a syn- 
chronized set of nodes has just fired while others are about to fire. This is addressed 
by adding to the definition nodes whose 0's are almost a Cycle apart. 

If all correct nodes in the system comprise a synchronized set of nodes then we 
say that the pulse_state is a synchronized _pulse_states of the system. The objective 
of the algorithm is hence to reach a synchronized_pulse_state of the system and to 
stay in such a state. The methodology to prove that our algorithm does exactly this 
will be to show firstly that a synchronized set of correct nodes stay synchronized. 
Secondly, we show that such synchronized sets of correct nodes incessantly join 
together to form bigger synchronized sets of nodes. This goes on until a synchro- 
nized set that encompasses all correct nodes in the system is formed. 

• The pulse_state of the system at real-time t is given by: 

pulse_state(t) = (0o(*), • • • , 4>n-i(t)) . 

• Let G be the set of all possible pulse_states of a system. 

• A set of nodes, S, is called synchronized at real-time t if 

Vpi,Pj € S, 4>i(t),(f)j(t) < cycle max , and one of the following is true: 

1. \4>i{t) -</>j(t)\ < a, or 

2. cycle min -cr < \(f>i(t)-(f>j(t)\ < cycle max and \<j>i(t-a)-<f)j(t-a)\ < 
a. 

• s S G is a synchronized_pulse_state of the system at real-time t if the set 
of correct nodes is synchronized at real-time t. 
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Definition 2.6. The Self-Stabilizing Pulse Synchronization Problem 

Convergence: Starting from an arbitrary system state, the system reaches a syn- 
chronized _pulse_state after a finite time. 

Closure: If s is a synchronized _pulse_state of the system at real-time to then V 
real-time t,t > t , 

1. pulse _state(t) is a synchronized _pulse_state, 

2. In the real-time interval [to, t] every correct node will invoke at most a 
single pulse if t — to > cycle m j n and will invoke at least a single pulse if 
t-t > cycle max . 

The second Closure condition intends to tightly bound the effective pulse invo- 
cation frequency within a priori bounds. This is in order to defy any trivial solution 
that could synchronize the nodes, but be completely unusable, such as instructing 
the nodes to invoke a pulse every a time units. Note that this is a stronger re- 
quirement than the "linear envelope progression rate" typically required by clock 
synchronization algorithms, in which it is only required that clock time progress as 
a linear function of real-time. 

3 The "Pulse Synchronization" Algorithm 

We now present the BIO-PULSE-SYNCH algorithm that solves the "Pulse Synchro- 
nization" problem defined in Definition 2.6, inspired by and following a neuro- 
biological analog. The refractory function describes the time dependency of the 
firing threshold. At threshold level the node invokes a pulse (fires) endogenously. 
The algorithm uses several sub-procedures. With the help of the SUMMATION pro- 
cedure, each node sums the pulses that it learns about during a recent time window. 
If this sum (called the Counter) crosses the current (time-dependent) threshold for 
firing, then the node will fire, i.e broadcasts its Counter value at the firing time. 
The exact properties of the time window for summing messages is determined by 
the message decay time in the PRUNE procedure (see Fig. 2). 

We now show in greater detail the elements and procedures described above. 

The refractory function 

The Cycle is the predefined time a correct node will count on its timer before in- 
voking an endogenous pulse. The refractory function, REF(t) : t — > {0..ra+l}, 
determines at every moment the threshold for invoking a new pulse. The refrac- 
tory function is determined by the parameters Cycle n, f, d and p. All correct 
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nodes execute the same protocol with the same parameters and have the same re- 
fractory function. The refractory function is shaped as a monotonously decreasing 
step function comprised of n + 2 steps, REF = (R n+ i, R n , Rq), where step 
Ri € M + is the time length on the node's timer of threshold level i. The refrac- 
tory function REF, starts at threshold level n + 1 and decreases with time towards 
threshold level 0. The time length of each threshold step is formulated in Eq. 1: 



Ri 



T^Cycle 
n-f 



Ri -R n +i -j^Cycle 
/+1 



I. ..n-f -I 

n-f...n (1) 



2d(l + p)- Kl ~^ i = n + l, 

Subsequent to a pulse invocation the refractory function is restarted at REF = 
n + 1. The node will then commence threshold level n only after measuring R n +\ 
time units on its timer. Threshold level (REF = 0) is reached only if exactly 
Cycle time units have elapsed on a node's timer since the last pulse invocation, 
following which threshold level n+1 is reached immediately. Hence, by definition, 
J27=i — Cycle. It is proven later in Lemma 4.2 that REF in Eq. 1 is consistent 
with this. 

The special step R n +i is called the absolute refractory period of the cycle. 
Following the neurobiological analogue with the same name, this is the first period 
after a node fires, during which its threshold level is in practice "infinitely high"; 
thus a node can never fire within its absolute refractory period. 

See Fig. 7 for a graphical presentation of the refractory function and its role in 
the main algorithm. 

The message sent when firing 

The content of a message M p sent by a node p, is the Counter, which represents 
the number of messages received within a certain time window (whose exact prop- 
erties are described in the appendix) that triggered p to fire. We use the notation 
Counter p to mark the local Counter at node p and Counter m p to mark the Counter 
contained in a received message M p sent by node p. 



3.1 The SUMMATION procedure 

A full account of the proof of correctness of the SUMMATION procedure is provided 
in the appendix. The SUMMATION procedure is executed upon the arrival of a 
new message. Its purpose is to decide whether this message is eligible for being 
counted. It is comprised of the following sub-procedures: 
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Upon arrival of the new message, the TIMELINESS procedure determines if 
the Counter contained in the message seems "plausible" (timely) with respect to 
the number of other messages received recently (it also waits a short time for such 
messages to possibly arrive). The bound on message transmission and process- 
ing time among correct nodes allows a node to estimate whether the content of 
a message it receives is plausible and therefore timely. For example, it does not 
make sense to consider an arrived message that states that it was sent as a result 
of receiving 2f messages, if less than / messages have been received during a re- 
cent time window. Such a message is clearly seen as a faulty node by all correct 
nodes. On the other, a message that states that it was sent as a result of receiv- 
ing 2/ messages, when 2f — 1 messages have been received during a recent time 
window does not bear enough information to decide whether it is faulty or not, as 
other correct nodes may have decided that this message is timely, due to receiving 
a faulty message. Such a message needs to be temporarily tabled so that it can be 
reconsidered for being counted in case some correct node sends a message within a 
short time, and which has counted that faulty message. Thus, intuitively, a message 
will be timely if the Counter in that messages is less or equal to the total number 
of tabled or timely messages that were received within a short recent time window. 
The exact length of the "recent" time window is a crucial factor in the algorithm. 
There is no fixed time after which a message is too old to be timely. The time for 
message exchange between correct nodes is never delayed beyond the network and 
processing delay. Thus, the fire of a correct node, as a consequence of a message 
that it received, adds a bounded amount of relay time. This is the basis for the time 
window within which a specific Counter of a message is checked for plausibility. 
Hence, a particular Counter of a message is plausible only if there is a sufficient 
number of other messages (tabled or not) that were received within a sufficiently 
small time window to have been relayed from one to the other within the bound 
on relaying between correct nodes. As an example, consider that the bound on 
the allowed relay interval of messages is taken to be 2d time units. Suppose that 
a correct node receives a message with Counter that equals k. That message will 
only be considered as timely if there are at least k + 1 messages that were received 
(including the last one) in the last k ■ 2d time window. This is the main criterion 
for being timely. On termination of the procedure the message is said to have been 
assessed. 

If a message is assessed as timely then the MAKE-ACCOUNTABLE procedure 
determines by how much to increment the Counter. It does so by considering the 
minimal number of recently tabled messages that were needed in order to assess the 
message as timely. This number is the amount by which the Counter is incremented 
by. A tabled message is marked as "uncounted" because the node's Counter does 
not reflect this message. Tabled messages that are used for assessing a message as 
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timely become marked as "counted" because the node's Counter now reflect these 
message as if they were initially timely. A node's Counter at every moment is 
exactly the number of messages that are marked as "counted" at that moment. 

The PRUNE procedure is responsible for the tabling of messages. A cor- 
rect node wishes to mark as counted, only those messages which considering the 
elapsed time since their arrival, will together pass the criterion for being timely at 
any correct nodes receiving the consequent Counter to be sent. Thus, messages that 
were initially assessed as timely are tabled after a short while. This is what causes 
the Counter to dissipate. After a certain time messages are deleted altogether (de- 
cayed). 

SUMMATIONS new message M p arrived at time i arr ) /* at node q */ 

if (TIMELINESS(M P , tarr) == "Afp is timely") then 

MAKE-ACCOUNTABLE(Mp) ; /* possibly increment Counter q */ 

PRUNE (t); 

Figure 1: The SUMMATION procedure 

The target of the SUMMATION procedure is formulated in the following two 
properties: 

Summation Properties: Following the arrival of a message from a correct node: 
PI: The message is assessed within d real-time units. 

P2: Following assessment of the message the receiving node's Counter is incre- 
mented to hold a value greater than the Counter in the message. 

The SUMMATION procedure satisfies the Summation Properties by the following 
heuristics: 

• When the Counter crosses the threshold level, either due to a sufficient counter 
increment or a threshold decrement, then the node sends a message (fires). 
The message sent holds the value of Counter at sending time. 

• The TIMELINESS procedure is employed at the receiving node to assess the 
credibility (timeliness) of the value of the Counter contained in this message. 
This procedure ensures that messages sent by correct nodes with Counter less 
than n will always be assessed as timely by other correct nodes receiving this 
message. 

• When a received message is declared timely and therefore accounted for 
it is stored in a "counted" message buffer ("Counted Set"). The receiving 
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node's Counter is then updated to hold a value greater than the Counter in 
the message by the MAKE- ACCOUNTABLE procedure. 

• If a message received is declared untimely then it is temporarily stored in an 
"uncounted" message buffer ("Uncounted Set") and will not be accounted 
for at this stage. Over time, the timeliness test of previously stored timely 
messages may not hold any more. In this case, such messages will be moved 
from the Counted Set to the Uncounted Set by the PRUNE procedure. 

• All messages are deleted after a certain time-period (message decay time) by 
the PRUNE procedure. 

Definitions and state variables: 

Counter: an integer representing the node's estimation of the number of timely 
firing events received from distinct nodes within a certain time window. Counter is 
updated upon receiving a timely message. The node's Counter is checked against 
the refractory function whenever one of them changes. The value of Counter is 
bounded and changes non-monotonously; the arrival of timely events may increase 
it and the decay/untimeliness of old events may decrease it. 

Stored message: is a basic data structure represented as (S p , t arr ) and created upon 
arrival of a message M p . S p is the id (or signature) of the sending node p and t arr 
is the local arrival time of the message. We say that two stored messages, (S p , ti) 
and (S q , t2), are distinct if p 7^ q. 

Counted Set (CS): is a set of distinct stored messages that determine the cur- 
rent value of Counter. The Counter reflects the number of stored messages in the 
Counted Set. A stored message is accounted for in Counter, if it was in CS when 
the current value of Counter was determined. 

Uncounted Set (UCS): is a set of stored messages, not necessarily distinct, that 
have not been accounted for in the current value of Counter and that are not yet 
due to decay. A stored message is placed (tabled) in the UCS when its message 
clearly reflects a faulty sending node (such as when multiple messages from the 
same node are received) or because it is not timely anymore. 

Retired UCS (RUCS): is a set of distinct stored messages not accounted for in the 
current value of Counter due to the elapsed local time since their arrival. These 
stored messages are awaiting deletion (decaying). 
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The CS and UCS are mutually exclusive and together reflect the messages re- 
ceived from other nodes in the preceding time window. Their union is denoted the 
node's Message_Pool. 

tsendMp- denotes the local-time at which a node p sent a message M p . An equiv- 
alent definition of t sen d m p is the local-time at which a receiving node p is ready to 
assess whether to send a message consequent to the arrival and processing of some 
other message. 

MessageAge(t, q, p): is the elapsed time, at time t, on a node g's clock since 
the most recent arrival of a message from node p, which arrived at local-time t arr . 
Thus, its value at node q at current local-time t is given by t — t arr , where M p is 
the most recent message that arrived from p. If no stored message is held at q for p 
then MessageAge(t,q,p) = oo. 

C SAge(t): denotes, at local-time t, the largest MessageAge(t, q, . . .) among 
the stored messages in CS of node q. 

( l+£ )fc+l i 

r: denotes the function r(k) = 2d(l + p) 1 ~i+ p . 

The set of procedures used by the summation procedure (at node q): 



The following procedure moves and deletes obsolete stored 
messages. It prunes the CS to hold only stored messages such that 
a message sent holding the resultant Counter will be assessed as 
timely at any correct node receiving the message. 

PRUNE {t) /* at node q */ 

• Delete from RUCS all entries (S p ,t) whose MessageAge(t,q,p) > 
r(n + 2); 

• Move to RUCS, from the Message_Pool, all stored messages (S p ,t) 
whose MessageAge(t,q,p) > r(n + 1); 

• Move to UCS, from CS, stored messages, beginning with the 
oldest, until: CSAge(t) < r(k - 1), where fe = max[l, \\CS\\]; 

• Set Counter := \\CS\\; 

Figure 2: The PRUNE procedure 
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We say that M p has been assessed by q, once the following procedure 
is completed. A message M p , is timely at local time t arr at node q 
once it is declared timely by the procedure, i.e. 1: whether the 
Counter in the message is within its valid range; 2: whether the 
sending node has recently sent a message, in which only the latest 
is considered; 3: whether enough messages have been received 
recently to support the credibility of the Counter in the message. 

TIMELINESS (M p ,t ar r) /* at node q */ 

/* check if Counter is valid */ 
Timeliness Condition 1: 

If (0 < Counter m p < n — 1) Then 

Create a new stored message (S p ,t arr ) and insert it into UCS; 
Else 

return "AI P is not timely"; 

/* if an older message from same node already exists then must be 
a faulty node. Delete all its entries but the latest. */ 
Timeliness Condition 2: 

If (3(S p ,t), s.t. t^t ar r, in Message_Pool U RUCS) Then" 
delete from Message_Pool all {S p ,t ) , where t' ^ t arr ; 
return "M p is not timely"; 

/* check if Counter m p seems credible with respect to the 
Message_Pool */ 
Timeliness Condition 3: 

Let k denote Counter m v - 

If (at some local-time t in the interval [t arr , t arr + d(l + p)] : 
\\{(S r ,t')\(S r ,t') e Message_Pool,MessageAge(t,q,r) < r(fc + l)}|| > fc + 1) Then* 

return "M p is timely"; 
Else 

return "M p is not timely"; 



"We assume no concomitant messages are stamped with the exact same arrival times at a correct 
node. We assume that one can uniquely identify messages. 

fc We assume the implementation can assess these conditions within the time window. 



Figure 3: The TIMELINESS procedure 



15 



This procedure moves stored messages from UCS into CS and updates 
the value of Counter. This is done in case the arrival of a new 
timely message M v , has made previously uncounted stored messages 
eligible for being counted. 

MAKE-ACCOUNTABLE (M p ) /* at node q */ 

• Move the max[l, (Counter m p — Counter q + 1)] most recent distinct 
stored messages from UCS to CS; 

• Set Counter := ||CS||; 

Figure 4: The MAKE- ACCOUNTABLE procedure 



This procedure causes the effective cycle of the node to be reset, 
meaning that the REF function starts the cycle from the highest 
threshold level again and down to threshold level . 

CYCLE-RESET () /* at node g */ 

• Restart REF at REF := n+1; 



Figure 5: The CYCLE-RESET procedure 



We now cite the main theorems of the SUMMATION procedure. The proofs are 
given in the appendix. 

Theorem 1. Any message, M p , sent by a correct node p will be assessed as timely 
by every correct node q. 

Lemma 3.1. Following the arrival of a timely message M p , at a node q, then at 
time t senc iM q i Counter q > CounterM p - 

Theorem 2. The SUMMATION procedure satisfies the Summation Properties. 

Proof. Let p denote a correct node that sends M p . Theorem 1 ensures that M p 
is assessed as timely at every correct node. Lemma 3.1 ensures that the value of 
Counter will not decrease below Counter m p + 1 until local-time t sen< iM p > thereby 
satisfying the Summation Properties. □ 
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3.2 The event driven "pulse synchronization" algorithm 

Fig. 6 shows the main algorithm. Fig. 7 illustrates the mode of operation of the 
main algorithm. 

BIO-PULSE-SYNCH(n, /, Cycle) /* at node q */ 

• It is assumed that all the parameters and variables are 
verified to be within their range of validity. 

• t is the local-time at the moment of executing the 
respective statement. 

if (a new message M p arrives at time t a -rr) then 
SUMMATION ( (M p , t arr ) ) ; 
if (Counter q > REF(t)) then 

Broadcast Counter q to all nodes; /* invocation of the 

Puise */ 

CYCLE-RESET ( ) ; 

if (change in threshold level according to REF) then 
PRUNE (t) ; 

if (Counter q > REF(t)) then 

Broadcast Counter q to all nodes; /* invocation of the 

Pulse */ 

CYCLE-RESET ( ) ; 



Figure 6: The event driven BIO-PULSE-SYNCH algorithm 




Figure 7: Schematic example of the mode of operation of BIO-PULSE-SYNCH: (a.) The node's 
Counter (the summed messages) does not cross the threshold during the cycle, letting the refractory 
function reach zero and consequently the node fires endogenously. (b.) Sufficient messages from 
other nodes are received in time window for the Counter to surpass the current threshold, conse- 
quently the node fires early and resets its cycle. 
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3.3 A Reliable-Broadcast Primitive 

In the current subsection we show that the BIO-PULSE-SYNCH algorithm can 
also operate in networks in which Byzantine nodes may exhibit true two-faced 
behavior. This is done by executing in the background a self-stabilizing Byzan- 
tine reliable-broadcast-like primitive, which assumes no synchronicity whatsoever 
among the nodes. It has the property of relaying any message received by a correct 
node. Hence, this primitive satisfies the broadcast assumption of Definition 2.2 by 
supplying a property similar to the relay property of the reliable-broadcast primi- 
tive in [31]. That latter primitive assumes a synchronous initialization and can thus 
not be used as a building block for a self-stabilizing algorithm. 

In [7] we presented the INITIATOR- ACCEPT primitive. We say that a node does 
an I-accept of a message m sent by some node p (denoted (p, m)) if it accepts that 
this message was sent by node p. 

The Initiator- Accept primitive essentially satisfies the following two prop- 
erties (rephrased for our purposes): 

IA-1A (Correctness) If all correct nodes invoke INITIATOR- ACCEPT (p, m) within 
d real-time of each other then all correct nodes I-accept (p, m) within 2d real- 
time units of the time the last correct node invokes the primitive INITIATOR- ACCEPT (j>, 

IA-3A (Relay) If a correct node q I-accepts (p, m) at real-time t, then every correct 
node q' I-accepts (p, m), at some real-time t', with \t — t'\ < 2d. 

The Initiator-Accept primitive requires a correct node not to send two suc- 
cessive messages within less than 6d real-time of each other. Following the BIO- 
PULSE-SYNCH algorithm (see Timeliness Condition 2, in the TIMELINESS pro- 
cedure), non-faulty nodes cannot fire more than once in every 2d(l + p) ■ n > 6d 
real-time interval even if the system is not coherent, which thus satisfies this re- 
quirement. 

The use of the INITIATOR- ACCEPT primitive in our algorithm is by execut- 
ing it in the background. When a correct node wishes to send a message it does 
so through the primitive, which has certain conditions for I-accepting a message. 
Nodes may also I-accept messages that where not sent or received through the 
primitive, if the conditions are satisfied. In our algorithm nodes will deliver mes- 
sages only after they have been I-accepted (also for the node's own message). From 
[IA-1A] we get that all messages from correct nodes are delivered within 3d real- 
time units subsequent to sending. From [IA-3A] we have that all messages are 
delivered within 2d real-time units of each other at all correct nodes, even if the 
sender is faulty. Thus, we get that the new network delay d = 3d. Hence, the cost 
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of using the INITIATOR-ACCEPT primitive is an added 2d real-time units to the 
achieved pulse synchronization tightness which hence becomes a = d = 3d. 



4 Proof of Correctness of BIO-PULSE-SYNCH 

In this section we prove Closure and Convergence of the BIO-PULSE-SYNCH 
algorithm. In the first subsection, 4. 1 , we present additional notations that facilitate 
the proofs. In the second subsection, 4.2, we prove Closure and in the third, 4.3, 
we prove Convergence. 

The proof that BIO-PULSE-SYNCH satisfies the pulse synchronization problem 
follows the steps below: 

Subsection 4.1 introduces some notations and procedures that are for proof 
purposes only. One such procedure partitions the correct nodes into disjoint sets of 
synchronized nodes ("synchronized clusters"). 

In Subsection 4.2 (Lemma 4.4), we prove that "synchronized clusters" once 
formed stay as synchronized sets of nodes, this implies that once the system is in a 
synchronized_pulse_state it remains as such (Closure). 

In Subsection 4.3 (Theorem 5), we prove that within a finite number of cycles, 
the synchronized clusters repeatedly absorb to form ever larger synchronized sets 
of nodes, until a synchronized_pulse_state of the system is reached (Convergence). 

Note that the the synchronization tightness, a, of our algorithm, equals d. 

It may ease following the proofs by thinking of the algorithm in the terms of 
non-liner dynamics, though this is not necessary for the understanding of any part 
of the protocol or its proofs. We show that the state space can be divided into 
a small number of stable fixed points ("synchronized sets") such that the state of 
each individual node is attracted to one of the stable fixed points. We show that 
there are always at least two of these fixed points that are situated in the basins of 
attraction ("absorbance distance") of each other. Following the dynamics of these 
attractors, we show that eventually the states of all nodes settle in a limit cycle in 
the basin of one attractor. 



4.1 Notations, procedures and properties used in the proofs 

First node in a synchronized set of nodes S, is a node of the subset of nodes that 
"fire first" in S that satisfies: 

"First node in 5" — < ™ n ^l* e max{(fo(i)|node i e S, <j>i(t) < a}} 3i € 5s. t. 4>i(i) < a 
1 min{z|« E max{^(i)|node i G S}} otherwise. 
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Equivalently, we define last node: 

"Last node in 5" — I max ^l* e min{^j(t)|node i £ S, 4>i(t) > a}} 3i 6 S s.t. 4>i(t) > a 
1 max{i|i £ min{</>,(i)|node i G S}} otherwise. 

The second cases in both definitions serve to identify the First and Last nodes 
in case t falls in-between the fire of the nodes of the set. 

Synchronized Clusters 

At a given time t the nodes are divided into disjoint synchronized clusters in the 
following way: 

1. Assign the maximal synchronized set of nodes at time t as a synchronized 
cluster. In case there are several maximal sets choose the set that is harboring 
the first node of the unified set of all these maximal sets. 

2. Assign the second maximal synchronized set of nodes that are not part of the 
first synchronized cluster as a synchronized cluster. 

3. Continue until all nodes are exclusively assigned to a synchronized cluster. 

The synchronized cluster harboring the node with the largest (necessarily finite) 
4> among all the nodes is designated C\ . The rest of the synchronized clusters are 
enumerated inversely to the <fi of their first node, thus if there are m synchronized 
clusters then C m is the synchronized cluster whose first node has the lowest 4> 
(besides perhaps C\). Note that at most one synchronized cluster may have nodes 
whose actual (p differences is larger than a, as it can contain nodes that have just 
fired and nodes just about to fire. The definition of C\ implies that at the time the 
nodes are partitioned into synchronized clusters (time t above) it may be the only 
synchronized cluster in such a state. 

The clustering is done only for illustrative purposes of the proof. It does not 
actually affect the protocol or the behavior of the nodes. In the proof we "assign" 
the nodes to synchronized clusters at some time t. From that time on we consider 
the synchronized clusters as a constant partitioning of the nodes into disjoint syn- 
chronized sets of nodes and we follow the dynamics of these sets. Thus, once a 
node is exclusively assigned to some synchronized cluster it will stay a member 
of that synchronized cluster. We aim at showing that eventually all synchronized 
clusters become one synchronized set of nodes. Once such a clustering is fixated 
we ignore nodes that happen to fail and forthcoming recovering nodes. Our proof 
is based on the observation that eventually we reach a time window within which 
the permanent number of non-correct nodes at every time is bounded by / and 
during that window the whole system converges. 
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OBSERVATION 4.1. The synchronized clustering procedure assigns every correct 
node to exactly one synchronized set of nodes. 

OBSERVATION 4.2. Immediately following the synchronized clustering procedure 
no two distinct synchronized clusters comprise one synchronized set of nodes. 

We use the following definitions and notations: 

• Ci — synchronized cluster number i. 

• Hi — cardinality of Ci (i.e. number of correct nodes associated with synchronized 
cluster Ci). 

• c — current number of synchronized clusters in the current state; c > 1. 

• dist(a, b, t) = \<p a (t) — 4>b(t) | is the distance (4> difference) between nodes a and 
b at real-time t. 

• 4> Ci (t) — is the <p(t) of the first node in synchronized cluster Cj. 

• dist(Ci,Cj,t) = dist((j) Ci (t), <p Cj (t), t) at real-time t. 

If at real-time t there exists no other synchronized cluster C r , such that 4> Ci if) > 
4>c r (t) > 4>c (t), then we say that the synchronized clusters C, and Cj are adjacent 
at real-time t. 

We say that two synchronized clusters, Ci and Cj , have absorbed if their union 
comprises a synchronized set of nodes. If a node in Cj fires due to a message 
received from a node in Cj, then, as will be shown in Lemma 4.7, the inevitable 
result is that their two synchronized clusters absorb. The course of action from the 
arrival of the message at a node in Cj until Cj has absorbed with Ci is referred to 
as the absorbance of Cj by Cj. 

We refer throughout the paper to the fire of a synchronized cluster instead of 
referring to the sum of the fires of the individual nodes in the synchronized cluster. 
In Lemma 7.8 we prove that these two notations are equivalent. 

In Theorem 3 we show that we can explicitly determine a threshold value, 
ad(Ci), that has the property that if for two synchronized clusters Cj and Cj, 
distiCi, Cj,t) < ad(Ci) then Cj absorbs Cj. We will call that value the "absorbance 
distance" of Cj. 

DEFINITION 4.1. The absorbance distance, ad(Cj), of a synchronized cluster Ci, 
is 

f+TH 

ad{Ci)= Y, R a 
s=/+i 

real-time units. 
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Properties used for the proofs 



We identify and prove several properties; one property of the SUMMATION 
procedure (Property 1) and several properties of REF (Properties 2-7). These are 
later used to prove the correctness of the algorithm. 

Property 1: See the Summation Properties in Subsection 3.1. 
Property 2: Ri is a monotonic decreasing function of i, Ri > Ri+i, for 
i = 1 ... ri — 1. 



Property 3: R, L > 3d + ^ £j=i Rj, for i = 1 . . . n - f - 1. 
Property 4: R t > <x(l - p) + ^ 22,-, for i = 1 . . . n. 

Property 5: R n+1 > 2d(l + . 
Property 6: i?i + ■ • • + i? n +i = Cycie. 



Consider any clustering of n — / correct nodes into c > 1 synchronized clus- 
ters, in which f denotes the largest synchronized. Thus riji is the number of nodes 
in the largest synchronized cluster and is less or equal to n — / — 1. The number 
of nodes in the second largest cluster is less or equal to \ } n ^ J • 

Property 7: 

c f+n, n j> c 

i? 9 + y^i^g > CycJe , where ^ nj = n — f . (2) 

i=ijVi'9=/+i 3=1 _P j=i 

We require the following restriction on the relationship between Cycle, d, n 
and / in order to prove that Properties 3-4 hold: 

Restriction 1: 

l + P yn+3 i 



;i - P 2 )[(i - P )(f + 1) + 2(1 + P ) • 



Cycie > d l _ p — ^ . (3) 



We now prove that Properties 2-7 are properties of REF: 
Lemma 4.1. Properties 2-5 are properties of REF under Restriction 1. 
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Proof. The proof for Properties 2 and 5 follows immediately from the definition of 
REF inEq. 1. 

Note that Ri > Rj, for 1 < i < n — f — 1 and n — f < j < n. Moreover, 
for a = d, Property 4 is more restrictive than Property 3. Hence, for showing that 
Properties 3 and 4 are properties of REF it is sufficient to show that Rj (where 
n — f < j < n) satisfies Property 4: 



Cycle /±±R\n+3 _ j „ 

1_P , " 2d(l + p) • Vl .7 + ^ - - jZ-Cyde > [d(l - p) + -£-Crcfe](/ + 1) 

-Cycle- -?—{n- f)Cycle- —?-(n- f)Cycle 



1-/9 l-p" l + p 

(Id 

> [d(l - p)(/ + 1) + 2d(l + p) • ^ - ](n - /) 
r l-p(n-/) 2p 



(^)" +3 -l. 



-(n - /)]CycJe 



1-p l+p 

> - p)(/ + 1) + 2(1 + p) • ^g___](„ - /) 



(S) n+3 -i 



( 1 _p)(l +p )_2p(l-p) (Id^n+S.! 

\_ 2 V ycie > d[(l - p)(/ + 1) + 2(1 + p) • ] 

iz£_3 p + p 2 (l±£)n+3_i 

n ~\_ n2 — c y cle > d i(i - p)(f + 1) + 2(1 + p) • ] =► 



(l-p 2 )[(l-p)(/ + l) + 2(l + p) 



/ l+P \n.+3 i 



C,c te > „ if - i - T? ^- . 

This inequality is exactly satisfied by Restriction 1 and thus Eq. 1 satisfies 
Properties 3 and 4. 

Note that for p = 0, the inequality becomes Cycle > d ■ (/ + l)(n — /). □ 
Lemma 4.2. Property 6 is a property of REF. 
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Proof. 



R\ H + Rn+l — (Rl + • • • + Rn-f-l) + (Rn-f H + i? n ) + 

= (n-/- 1 )' I ^ J r + (/+1) 7+1 

1 recycle 

Cycle - — — + R ± - R n+1 - Cycle + R n+1 = Cycle 



1 - p n- } 1- p 

□ 

Lemma 4.3. Property 7 is a property of REF. 

Proof. We will prove that the constraint in Eq. 2 is always satisfied by the refrac- 
tory function in Eq. 1 . 

Note that Eq. 2 is a linear equation of the R t values of REF. We denoted 
riji to be the number of nodes in the largest synchronized cluster, following some 
partitioning of the correct nodes into synchronized clusters. We want to find what is 
the largest value of i such that R{ is a value with a non-zero coefficient in the linear 
equation Eq. 2. This value is determined by either the largest possible cluster, 
which may be of size n — / — 1 (in case all but one of the correct nodes are in 
one synchronized cluster 6 ), or by the second-largest possible cluster, which may 
be of size [ ^ \ (in case all correct nodes are in two possibly equally sized 
synchronized clusters). Thus the largest value of i such that R t is a value with 
a non-zero coefficient equals max[/ + [ ^^ J, n — / — 1] = n — / — 1, for 
n > 3/ + 1. 

y^— Cycle 

Thus, following Eq. 1, each of these Ri values equals — — . There are 
exactly n — f (not necessarily different) Ri values in Eq. 2. Hence, incorporating 
Eq. 1 into Eq. 2 reduces Eq. 2 to the linear equation: (n — /) • R4 > j^Cycle, 
where I < i < n — f — 1. It remains to show that Eq. 1 satisfies this constraint: 

1 Cycle 1 

(n-f)-Ri = (n - /) • = Cycle. 

n- J 1 - p 

□ 



4.2 Proving the Closure 

We now show that a synchronized set of nodes stays synchronized. This also im- 
plies that the constituent nodes of a synchronized clusters stay as a synchronized 

6 The case in which the n — f correct nodes are in one synchronized cluster implies the objective 
has been reached. 
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set of nodes, as a synchronized cluster is in particular a synchronized set of nodes. 
This proves the first Closure requirement of the "Pulse Synchronization" problem 
in Definition 2.6. 

Lemma 4.4. A set of correct nodes that is a synchronized set at real-time t' , re- 
mains synchronized Vt, t >t' . 

Proof. Let there be a synchronized set of nodes at real-time t'. From the defini- 
tion of a synchronized set of nodes, this set of nodes will stay synchronized as 
long as no node in the set fires. This is because the <fi difference between nodes 
(in real-time units) does not change as long as none of them fires. We there- 
fore turn our attention to the first occasion after t' at which a node from the set 
fires. Let us examine the extreme case of a synchronized set consisting of at 
least two nodes at the maximal allowed (j> difference; that is to say that at time t', 
dist(first_node, last_node, t') = a. Further assume that the first node in the set 
fires with a Counter=fe, (0 < k < n — 1), at some time t > t' at the very beginning 
of its threshold level k, and without loss of generality is also the first node in the 
set to fire after time t'. We will show that the rest of the nodes in the set will fire 
within the interval [t, t + a] and thus remains a synchronized set. 

Property 1 ensures that the last node's Counter will read at least k + 1 sub- 
sequently to the arrival and assessment of the first node's fire, since its Counter 
should be at least the first node's Counter plus 1. The proof of the lemma will be 
done by showing that right after the assessment of the first node's fire, the last node 
cannot be at a threshold higher than k + 1 and thus will necessarily fire. 

The proof is divided into the following steps: 

1. Show that when the first node is at threshold level k then the last node is at 
threshold level k + 1 or lower. 

2. Show that if the first node fires with a Counter=/c then due to Property 1 and 
Step 1 the last node will fire consequently. 

3. Show that the last node fires within a d real-time window of the first node, 
and as a result, the new distance between the first and last node is less than 
or equal to a. 

Observe that the extreme case considered is a worst case since if the largest <fi 
difference in the set is less than a then the threshold level of the last node may only 
be lower. The same argument also holds if the first node fires after its beginning 
of its threshold level k. Thus the steps of the proofs also apply to any intermediate 
node in the synchronized set and thus remains as a synchronized set of nodes. 
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Step 1: In this step we aim at calculating the amount of time on the last node's 
clock remaining until it commences its threshold level k, counting from the event 
in which the first nodes commences its threshold level k. By showing that this 
remaining time is less than the length of threshold level k + 1, as counted on the 
clock of the last and slowest node we conclude that this node must be at most at 
threshold level k + 1. The calculations are done on the slow node's clock. 

Assume the first node to be the fastest permissible node and the last one the 
slowest. Hence, when the first node's threshold level k commences, 

n+1 

T+~i> S Ri <» 

K i=k+l 

real-time units actually passed since it last fired. The last node "counted" this 
period as: 

, n+1 

y Ri . (6) 

y i=k+l 

The last node has to count on its clock, from the time that the first node fired, 
at most a (I — p) local-time units (max. <fi difference of correct nodes in a synchro- 
nized set as counted by the slowest node), and 

n+1 

^ ( 7 ) 

i=k+l 

in order to reach its own threshold level i As a result, the maximum local- 
time difference between the time the first node starts its threshold level k till the 
last node starts its own threshold level k as counted by the last node is therefore 
<t(1 + p) plus the difference Eq. 7 - Eq. 6, which yields 



1 n+1 1 n+1 „ n+1 

*{l-p) + ±±£ £ m-L-t Ri = a(l-p) + -^ £ R t . (8) 
1 + P i £&i l + Pi^+i 1 + P l ^+i 

Property 4 ensures that Rk+i is greater than Eq. 8 for < k < n— 1; thus when 
the first node commences threshold level k the last node must be at a threshold level 
that is less or equal to k + 1. 

Step 2: Let the first node fire as a result of its Counter equalling k at time t at 
threshold level k. In case that the last node receives almost immediately the first 
node's fire (and thus increments its Counter to at least k + 1 following Property 1), 
it must be at a threshold level that is less or equal to k + 1 (following Step 1) and 
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will therefore fire. All the more so if the first node's fire is received later, since the 
threshold level can only decrease in time before a node fires. 

Step 3: We now need to estimate the new distance between the first and last node 
in order to show that they still comprise a synchronized set. The last node assesses 
the first node's fire within d real-time units after the first node sent its message 
(per definition of d). This yields a distance of d(l — p) as seen by the last node, 
which equals the maximal allowed real-time distance, d (= a), between correct 
nodes in a synchronized set at real-time t', and thus they stay a synchronized set at 
time t'. □ 

Corollary 4.5. (Closure 1) Lemma 4.4 implies the first Closure condition. 

Lemma 4.6. ( Closure 2) As long as the system state is in a 
synchronized _pulse_state then the second Closure condition holds. 

Proof. Due to Lemma 4.4 the first node to fire in the synchronized set following 

its previous pulse, may do so only if it receives the fire from faulty nodes or if it 

fires endogenously. This may happen the earliest if it receives the fire from exactly 

/ distinct faulty nodes. Thus following Eq. 1 its cycle might have been shortened 
Cycle 

by at most / • real-time units. Hence, in case the first node to fire is also a 

fast node, it follows that cycle min = Cycle ■ (1 — p) — ■ Cycle ■ (1 — p) = n ~^j ■ 
Cycle ■ (1 — p) real-time units. A node may fire at the latest if it fires endogenously. 
If in addition it is a slow node then it follows that cycle max = Cycle ■ (1 + p) 
real-time units. 

Thus in any real-time interval that is less or equal to cycle m j n any correct node 
will fire at most once. In any real-time interval that is greater or equal to cycle max 
any correct node will fire at least once. This concludes the second closure condi- 
tion. □ 



4.3 Proving the Convergence 

The proof of Convergence is done through several lemmata. We begin by pre- 
senting sufficient conditions for two synchronized clusters to absorb. In Subsec- 
tion 4.3.1, we show that the refractory function REF ensures the continuous exis- 
tence of a pair of synchronized clusters whose unified set of nodes is not synchro- 
nized, but are within an absorbance distance and hence absorb. Thus, iteratively, 
all synchronized clusters will eventually absorb to form a unified synchronized set 
of nodes. 
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Lemma 4.7. (Conditions for Absorbance) Given two synchronized clusters, Ci 
preceding Cj, if: 

1. Ci fires with Counter-k, at real-time t Ci _fi res , where < k < f 

2. dist(Ci,Cj,t Ci _f ires ) < 2~2 g =k+i ~ T^p 1 2~2g=k+i^9 
then Ci will absorb Cj . 

Proof. The proof is divided into the following steps: 

1 . (a) If Ci fires before Cj , then Cj consequently fires. 

(b) Subsequent to the previous step: dist(Ci, Cj, ..) < 3d. 

2. Following the previous step, within one cycle the constituent nodes of the 
two synchronized clusters comprise a synchronized set of nodes. 

Step la: Let us examine the case in which Cj fires first at some real-time denoted 
t Ci _fi res , and in the worst case that Cj doesn't fire before it receives all of Cj's 
fire. All the calculations assume that at t c ._f ires , (\>c^ajires) has still not been 
reset to 0. Specifically, assume that the first node in Ci fired due to incrementing 
its Counter to k (0 < k < /) at the beginning of its threshold level k. Following 
Property 1 and Lemma 7.8 the nodes of Cj increment their Counters to k + raj after 
receiving the fire of Cj . Additionally, in the worst case, assume that the first node 
in Cj receives the fire of Cj almost immediately. We will now show that this fire is 
received at a threshold level < k + n j . 

We will calculate the upper-bound on the (f> of the first node in Cj at real-time 
t Ci _fi res , and hence deduce the upper-bound on its threshold level. Assume the 
nodes of Cj are fast and the nodes of Cj are slow. Should the nodes of Cj be faster, 
then the threshold level may only be lower. 
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We now seek to deduce the bound on C/s threshold level at the time of Cj's 
fire. Thus, following Eq. 9, at real-time t Ci ji res the </> of the first node in Cj is 
at most i-^— y^!! + i , i , „ -R<7. We assumed the worst case in which the constituent 

1— p {-~/g=K+L+ni y 

correct nodes of Cj are slow, thus these nodes have counted on their timers at least 

C 1 - P) ■ T^T^Zl+i+n^g = YTgtl+i+nRg time units since their last P ulse - 
Hence, the coiTect nodes of Cj are at real-time t Ci _f ires at most in threshold level 

k + rij. Should k < f or the fire of Cj be received at a delay, then this may only 

cause the threshold level at time of assessment of the fire from Cj to be equal or 

even smaller than k + ?ij. Thus, Lemma 4.4 and Property 1 guarantee that the first 

node in Cj will thus fire and that the rest of the nodes in both synchronized clusters 

will follow their respective first ones within a real-time units. 



Step lb: We seek to estimate the maximum distance between the two synchro- 
nized clusters following the fire of Cj. The first node in Cj will fire at the latest 
upon receiving and assessing the message of the last node in Cj. More precisely, 
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fire at the latest d real-time units following the fire of the last node in Cj , yield- 
ing a new dist(Ci, Cj, ..) of at most 2d real-time units regardless of the previous 
dist(C{, Cj, ..), rii, k and rij. The last node of Cj is at most at a distance of d 
from the first node of Cj therefore making the maximal distance between the first 
node of Q and the last node of Cj, at the moment it fires, equal 3d real-time units. 

Step 2: We will complete the proof by showing that after C, causes Cj to fire, the 
two synchronized clusters actually absorb. We need to show that in the cycle sub- 
sequent to Step 1, the nodes that constituted C-i and Cj become a synchronized set. 
Examine the case in which following Step 1, either one of the two synchronized 
clusters increment its Counter to k! and fires at the beginning of threshold level k' . 
We will observe the (f> of the first node to fire, denoted by 4>Urst_node-2nd-cycle- Fol- 
lowing the same arguments as in Step 1, all other nodes increment their Counters 
to k' + 1 after receiving this node's fire. Consider that this happens at the moment 
that this first node incremented its Counter to k' and fired, denoted hnd-cycle-fire- 
Below we compute, using Property 3, the lower bound on the <j> of the rest of the 

nodes at real-time t^nd- cycle- fire, denoted <fi 'other -nodesihnd- cycle- fire) • 



Mother— nodes(t2nd— cycle— fire) ^ ^lirstjnode— 2nd— cycle^p2nd— cycle— fire) 3d 
n+1 ^ n+1 
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In the worst case, the rest of the constituent nodes that were in Cj and Cj are 
slow nodes and thus, at real-time t 2n d-cycie-fire, counted: 
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time units since their last pulse. Due to Property 3 all these correct nodes 
receive the fire and increment their Counters to k' + 1 in a threshold level which is 
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less or equal to k' + 1 and will fire as well within d real-time units of the first node 
in the second cycle. □ 



Theorem 3. ( Conditions for Absorbance) Given two synchronized clusters, Cj pre- 
ceding Cj, if: 

1. Ci fires with Counter=k, at real-time t c ._fi res , where < k < f, and 

2. 3 t, t prev _ Cj _ fired < t < t^ji^, for which dist(d,Cj,t) < ad(Ci) 
then Ci will absorb Cj . 

Proof. Denote t prev _ Cj _fi re d the real-time at which Cj previously fired before time 
tajires- Given that at some time t, where t prev _ Cj _ fired < t < t Ci _ fires , dist(Ci,Cj,t) < 
ad(Ci), we wish to calculate the maximal possible distance between the two syn- 
chronized clusters at real-time t Ci _ji res , the time at which Cj fires with Counter=fe, 
where < k < f. 

Under the above assumptions, the maximal possible distance at real-time t c% _fi res 
is obtained when k = f and when at time t prev _ c _fi re d the distance between Cj 
and Cj was exactly ad(Cj), i.e dist(Ci,Cj,t prev _ Cj _fi re d) = ad(Cj). The upper 
bound on dist(Ci,Cj,t Ci _u res ) takes into account that from C-s previous real- 
time firing time, t prev _ Ci _fi rec i, and until real-time t Ci _fi res , the nodes of Cj were 
fast and that from real-time t prev _ Cj _fi re d and until t Ci ji res , the nodes of Cj were 
slow. Thus the bound on dist(Ci,Cj,t Ci _fi res ) becomes the real-time difference 
between these: 
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Eq. 12 is the upper bound on the distance between the two synchronized clus- 
ters at real-time t c ._f ires , thus following Lemma 4.7, the two synchronized clusters 
absorb. □ 



4.3.1 Convergence of the Synchronized Clusters 

In the coming subsection we look at the correct nodes as partitioned into synchro- 
nized clusters (at some specific time). Observation 4.2 ensures that no two of these 
synchronized clusters comprise one synchronized set of nodes. The objective of 
Theorem 4 is to show that within finite time, at least two of these synchronized 
clusters will comprise one synchronized set of nodes. Specifically, we show that in 
any state that is not a synchronized_pulse_state of the system, there are at least two 
synchronized clusters whose unified set of nodes is not a synchronized set but that 
are within absorbance distance of each other, and consequently they absorb. Thus, 
eventually all synchronized clusters will comprise a synchronized set of nodes. 

We claim that if the following relationship between REF and Cycle is sat- 
isfied, then absorbance (of two synchronized clusters whose unified set is not a 
synchronized set), is ensured irrespective of the states of the synchronized clusters. 
Let Cji denote the largest synchronized cluster. The theorem below, Theorem 4, 
shows that for a given clustering of n — / correct nodes into c > 1 synchronized 
clusters and for n, /, Cycle and REF that satisfy 

c n j' 

ad(Cj) + -—jy g > — ■ Cycle (13) 

there exist at least two synchronized clusters, whose unified set is not a synchro- 
nized set of nodes, that will eventually undergo absorbance. 

Note that Eq. 13 is derived from Property 7 (Eq. 2): 

Eq. 2 derives the following equation (since the R g values are non-negative), 

c f+ n j n j' 

E E^ + r^E^^r^- c ^ de • ^ 

j=ljVi'9=/+i H 9=1 ^ 

Incorporating the absorbance distance of Definition 4. 1 into Eq. 14 yields ex- 
actly Eq. 13. We use Eq. 13 in Theorem 4 instead of Eq. 2 (Property 7) for read- 
ability of the proof. 

Theorem 4. (Absorbance) Assume a clustering ofn — f correct nodes into c > 1 
synchronized clusters at real-time to- Further assume that Eq. 13 holds for the 
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resulting clustering. Then there will be at least one synchronized cluster that will 
absorb some other synchronized cluster by real-time to + 2 • cycle. 

Proof. Note that following the synchronized cluster procedure, the unified set of 
the two synchronized clusters that will be shown to absorb, are not necessarily a 
synchronized set of nodes at time to- Assume without loss of generality that Cy is 
the synchronized cluster with the largest number of nodes, consequent to running 
the clustering procedure. Exactly one out of the following two possibilities takes 
place at to: 

1. 3i (1 < i < c), such that dist(Cj, C( i+1 )( mod c ), to) < ad(C{). 

2. Vi (1 < i < c,i / j'),dist(Ci,C( i+ ix modc ),t ) > ad{d). 

Consider case 1. Following the protocol, Cj must fire within Cycle local-time 
units of to- Observe the first real-time, denoted tj, at which d fires subsequent to 
real-time to- Assume that k > is the number of distinct inputs that causes the 
Counter of at least one node in d to reach the threshold and fire (not counting the 
fire from nodes in d itself). If k > f then at least one correct node outside of Q 
caused some node in Cj to fire. This correct node must belong to some synchro- 
nized cluster which is not Cj . We denote this synchronized cluster C x as its identity 
is irrelevant for the sake of the argument. We assumed that at least one node in d 
fired due to a node in C x . Following Lemma 4.4 the rest of the nodes in Cj. will fol- 
low as well, as a synchronized cluster is in particular a synchronized set of nodes. 
This yields a new dist(C x , Cj, ..) of at most 3d. Following the same arguments 
as in Step 2 of Lemma 4.7, C x and d hence absorb. Therefore the objective is 
reached. Hence assume that k < f and that d did not absorb with any preced- 
ing synchronized cluster. Thus, the last real-time that C7j + iv m0( j c \ fired, denoted 
t Ci+1 -fired, was before or equal to real-time t , i.e. t Ci+1 ~fired < to < U and 
dist(d, C(i+i)(mod c),to) < ad(d). By Theorem 3, d will absorb C {m) ( mod c) . 

Consider case 2. We do not assume that dist(dj>, CVj/ +1 y mod c ), to) > ad(Cji). 
Assume that there is no absorbance until Cj> fires (otherwise the claim is proven). 
Let ty denote the real-time at which the first node in Cy fires, at which 4> c ., (ty ) = 
0. There are two possibilities at ty: 

2a. 3i(l <i < c), such that at ty, dist(di, C^ + iv mo( j c )ity) < ad(d)- 

2b. Vi(l <i<c,i^ f),dist{d,C {i+1){mod c) ,ty) > ad(d)- 

Consider case 2a. This case is equivalent to case 1. The last real-time that 
C(i+i)(mod c) fired, denoted tc i+1 -fi T edi was before or equal to real-time ty. De- 
note U the real-time at which the first node of d fires. Thus, tc i+1 - fired < 
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tjt < ti and dist(Ci, C(j+i)( mo d c ); *o) < ad(Ci). By Theorem 3, Cj will ab- 
sorb C( i+1 )( mod c ). 

Consider case 2b. We wish to calculate <j> Cj , +1 (tj') an d from this deduce the 
upper bound on the threshold level of the first node in C(y +1 )( mod c ) at real-time 
ty . We first want to point out that 

c 

<^y + i(V)> £ orf(Cj)- (15) 

This stems from the fact that Cy has just fired and that Cy and C(j/ +1 )( mod c ) are 
adjacent synchronized clusters which implies that 

Vi(l <i<c,i ^ j' + l),4>c jl+1 (tf) > 4> Cl {tj>). 

Recall that C , (fy) = 0. From the case considered in 2b we have that 

Vi(l < i < c,i^f), dist{Ci, C (i+1) ( modc ),t f ) > ad(d). 

Thus Eq. 15 follows. Following Eq. 13 and Eq. 15 we get: 

0c,, +1 (*j')> ^ ad(C,) > ^ • CycJe --^J^,. (16) 

In the worst case the nodes of C(j/ +1 )( moc j c \ are slow. Thus at real-time tj> 
they have measured, from their last pulse, at least (1 — p) ■ Cj , +1 (tj 1 ) = (1 — p) • 
[-^ ■ Cycle - -^- p Y? g Li R g\ = T^=l jl+1 R g local-time units. Thus, following 
Property 1, the first node in CVj/ +1 y m0( j c \ receives the fire from Cy and increment 
its Counter to at least ny in a threshold level which is less or equal to ny and 
will thus fire as well. Following Lemma 4.4 the rest of the synchronized cluster 
will follow as well. This yields a new dist(Cj/, Cy/ +1 v mod c \, ..) of at most 3d. 
Following the same arguments as in Step 2 of Lemma 4.7, Cy and Cf ? v +1 v ino( j a 
hence absorb. 

Thus at least two synchronized clusters will absorb within 2 • cycle of to which 
concludes the proof. □ 

The following theorem assumes the worst case of n = 3/ + 1. 

Theorem 5. (Convergence) Within at most 2(2 / + 1) • cycle real-time units the 
system reaches a synchronized _pulse_state. 

Proof. Assume that n = 3/ + 1. Thus, the maximal number of synchronized 
clusters is 2/ + 1, and since following Theorem 4 at least two synchronized clusters 
absorb in every two cycles we obtain the bound. □ 
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5 Analysis of the Algorithm and Comparison to Related 
Algorithms 



The protocol operates in two epochs: In the first epoch there is no limitations on 
the number of failures and faulty nodes. In this epoch the system might be in any 
state. In the second epoch there are at most / nodes that may behave arbitrarily at 
the same time, from which the protocol may start to converge. Nodes may fail and 
recover and nodes that have just recovered need time to synchronize. Therefore, 
we assume that eventually we have a window of time within which the turnover be- 
tween faulty and non-faulty nodes is sufficiently low and within which the system 
inevitably converges (Theorem 4). 

Authentication and fault ratio: The algorithm does not require the power 
of unforgeable signatures, only an equivalence to an authenticated channel is re- 
quired. Note that the shared memory model ([13]) has an implicit assumption that 
is equivalent to an authenticated channel, since a node "knows" the identity of the 
node that wrote to the memory it reads from. A similar assumption is also implicit 
in many message passing models by assuming a direct link among neighbors, and 
as a result, a node "knows" the identity of the sender of a message it receives. 

Many fundamental problems in distributed networks have been proven to re- 
quire 3/ + 1 nodes to overcome / concurrent Byzantine faults in order to reach a 
deterministic solution without authentication [18, 24, 11, 10]. We have not shown 
this relationship to be a necessary requirement for solving the "Pulse Synchroniza- 
tion" problem but the results for related problems lead us to believe that a similar 
result should exist for the "Pulse Synchronization" problem. 

There are algorithms that have no lower bound on the number of nodes required 
to handle / Byzantine faults, but unforgeable signatures are required as all the 
signatures in the message are validated by the receiver [11]. This is costly time- 
wise, it increases the message size, and it introduces other limitations, which our 
algorithm does not have. Moreover, within the self-stabilizing paradigm, using 
digital signatures to counter Byzantine nodes exposes the protocols to "replay- 
attack" which might empty its usefulness. 

Convergence time: We have shown in [5] that self-stabilizing Byzantine clock 
synchronization can be derived from self-stabilizing Byzantine pulse synchroniza- 
tion. Conversely, self-stabilizing Byzantine clock synchronization can be used to 
trivially produce self-stabilizing Byzantine pulse synchronization. Thus the two 
problems are supposedly equally hard. The only self-stabilizing Byzantine clock 
synchronization algorithms besides [5] are found in [13]. The randomized self- 
stabilizing Byzantine clock synchronization algorithm published there synchro- 
nizes in M ■ 2 2 ( n ~^ steps, where M is the upper bound on the clock values held 
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by individual processors. The algorithm uses message passing, it allows transient 
and permanent faults during convergence, requires at least 3/ + 1 processors, but 
utilizes a global pulse system. An additional algorithm in [13], does not use a 
global pulse system and is thus partially synchronous similar to our model. The 
convergence time of the latter algorithm is 0((n — f)n 6 ( n ~^). This is drastically 
higher than our result, which has a cycle length of 0(f 2 ) ■ d time units and con- 
verges within 2(2/ + 1) cycles. The convergence time of the only other correct 
self-stabilizing Byzantine pulse synchronization algorithm [9] has a cycle length 
of 0(f) ■ d time units and converges within 6 cycles. 

Message and space complexity: The size of each message is 0(logn) bits. 
Each correct node multicasts exactly one message per cycle. This yields a message 
complexity of at most n messages per cycle. The system's message complex- 
ity to reach synchronization from any arbitrary state is at most 2n(2f + 1) mes- 
sages per synchronization from any arbitrary initial state. The faulty nodes cannot 
cause the correct nodes to fire more messages during a cycle. Comparatively, the 
self-stabilizing clock synchronization algorithm in [13] sends n messages during 
a pulse and thus has a message complexity of 0(n(n — f)n^ n ~^). This is sig- 
nificantly larger than our message complexity irrespective of the time interval be- 
tween the pulses. The message complexity of the only other correct self-stabilizing 
Byzantine pulse synchronization [9] equals 0(n 3 ) per cycle. 

The space complexity is 0(n) since the variables maintained by the processors 
keep only a linear number of messages recently received and various other small 
range variables. The number of possible states of a node is linear in n and the node 
does not need to keep a configuration table. 

The message broadcast assumptions, in which every message, even from a 
faulty node, eventually arrives at all correct nodes, still leaves the faulty nodes 
with certain powers of multifaced behavior since we assume nothing on the order 
of arrival of the messages. Consecutive messages received from the same source 
within a short time window are ignored, thus, a faulty node can send two concomi- 
tant messages with differing values such that two correct nodes might receive and 
relate to different values from the same faulty node. 

Tightness of synchronization: In the presented algorithm, the invocation of 
the pulses of the nodes will be synchronized to within the bound on the relay time of 
messages sent and received by correct nodes. In the broadcast version, this bound 
on the relay time equals d real-time units. Note that the lower bound on clock 
synchronization in completely connected, fault-free networks [23] is d(l — 1/n). 
We have shown in Section 3.3 how the algorithm can be executed in non-broadcast 
networks to achieve a synchronization tightness of a = 3d. Comparatively, the 
clock synchronization algorithm of [1 1] reaches a synchronization tightness typical 
of clock synchronization algorithms of d(l + p) + 2p(l + p) ■ R, where R is the 
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time between re-synchronizations. The second Byzantine clock synchronization 
algorithm in [13] reaches a synchronization tightness which is in the magnitude of 
(n — f) ■ d(l + p). This is significantly less tight than our result. The tightness 
of the self-stabilizing Byzantine pulse synchronization in [9] equals 3d real-time 
units. 

Firing frequency bound: The firing frequency upper bound during normal 

steady-state behavior is around twice that of the endogenous firing frequency of 

Cycle 

the nodes. This is because cycle m j n > — ^ — • This bound is influenced by the 
fraction of faulty nodes (the sum of the first / threshold steps relative to Cycle). 
For n = 3/ + 1 this translates to w \ Cycle. Thus, if required, the firing frequency 
bound can be closer to the endogenous firing frequency of 1 • Cycle if the fraction 
of faulty nodes is assumed to be lower. For example, for a fraction of fault nodes 
of / = jg, the lower bound on the cycle length, cycle m j n , becomes approximately 
8/9 that of the endogenous cycle length. cycle max = Cycle ■ (1 + p) real-time units. 



6 Discussion 

We developed and presented the "Pulse Synchronization" problem in general, and 
an efficient linear-time self-stabilizing Byzantine pulse synchronization algorithm, 
BIO-PULSE-SYNCH, as a solution in particular. The pulse synchronization problem 
poses the nodes with the challenge of invoking regular events synchronously. The 
system may be in an arbitrary state in which there can be an unbounded number of 
Byzantine faults. The problem requires the pulses to eventually synchronize from 
any initial state once the bound on the permanent number of Byzantine failures is 
less than a third of the network. The problem resembles the clock synchronization 
problem though there is no "value" (e.g. clock time) to agree on, rather an event in 
time. Furthermore, to the best of our knowledge, the only efficient self-stabilizing 
Byzantine clock synchronization algorithm assumes a background pulse synchro- 
nization module. 

The algorithm developed is inspired by and shares properties with the lobster 
cardiac pacemaker network; the network elements (the neurons) fire in tight syn- 
chrony within each other, whereas the synchronized firing pace can vary, up to a 
certain extent, within a linear envelope of a completely regular firing pattern. 

A number of papers have recently postulated on the similarity between ele- 
ments connected with biological robustness and design principles in engineering 
[1, 19]. In the current paper we have observed and understood the mechanisms 
for robustness in a comprehensible and vital biological system and shown how to 
make specific use of analogies of these elements in distributed systems in order to 
attain high robustness in a practical manner. The same level of robustness has not 
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been practically achieved earlier in distributed systems. We postulate that our re- 
sult elucidates the feasibility and adds a solid brick to the motivation to search for 
and to understand biological mechanisms for robustness that can be earned over to 
computer systems. 

The neural network simulator SONN ([29]) was used in early stages of de- 
veloping the algorithm for verification of the protocol in the face of probabilistic 
faults and random initial states. It is worth noting that the previous pulse synchro- 
nization procedure found in [5] was mechanically verified at NASA LaRC ([25]) 
which greatly facilitated uncovering its flaw. A natural next step should thus be 
to undergo simulation and mechanical verification of the current protocol that can 
mimic a true distributed system facing transient and Byzantine faults. 
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7 Appendix 

Proof of correctness of the summation procedure: 



Lemma 7.1. For k G N, k > 0, 



Proof. 



r{k) ■ + 2d(l + p) = r(k + 1) . 

1 - p 



• + 2d ^ + p) = t 2 ^ 1 + p) (±L-i ] ■ w + 2d{1 + p) 

k fc+1 

[2d(l+p)^(i±^) i ].i±^+2d(l+p) = [2d(l+p)Y,{\ ±£ ) l \+m+P) 
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□ 
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Lemma 7.2. Let a correct node q receive a message M p from a correct node p at 
local-time t arr . For every one ofp's stored messages (S r , t') that is accounted for 
in Counter m p , then at q,from some time t in the local-time interval [t arr ,t arr + 
d(l + p)} and at least until the end of the interval: 

MessageAge(t, q, r) < r(CounterM p + 1) • 

Proof. Following the PRUNE procedure at p, the oldest of its stored messages ac- 
counted for in Counter m p was at most r(CounterM p ) time units old on p's clock 
at the time it sent M p . This oldest stored message could have arrived at q, 6(1 + p) 
local-time units on q's clock, prior to its arrival at p. Within this time p should also 
have received all the messages accounted for in M p . Another it (I + p) local-time 
units could then have passed on q's clock until M p was sent. M p could have arrived 
at q, 6(1 + p) time units on q's clock after it was sent by p. By this time q would 
also have received all the messages that are accounted for in M p , irrespective if 
q had previous messages from the same nodes. Another 7r(l + p) time units can 
then pass on q's clock until all messages are processed. Thus, in the worst case 
that node p is slow and node q is fast and by Lemma 7.1, for every stored message 
accounted for in Counter m v , 3t € [t arr + d(l + p)], we have: 

Message Age(t, q, r) < MessageAge(t arr + d(l + p),q, r) 

< T(Counter Mv ) ■ \^ + 6(1 + p) + vr(l + p) + 6(1 + p) + vr(l + p) 
1 - p 

= r(Counter M ) ■ + - + 2d(l + p) = r(Counter Mv + 1) . 
1 — p 

□ 

Lemma 7.3. The Counter of a correct node cannot exceed n and a correct node 
will not send a Counter that exceeds n — 1. 

Proof. There can be at most n distinct stored messages in the CS of a correct node 
hereby bounding the Counter by n. 

For a correct node to have a Counter that equals exactly n it needs its own 
stored message to be in its CS, as a consequence of a message it sent. Consider the 
moment after it sent this message, say before the node's Counter reached n, that 
is accounted for in its CS. This message was concomitant to its pulse invocation 
and cycle reset. The node assesses its own message at most d(l + p) local-time 
units after sending it thus, following the PRUNE procedure, its own stored message 
will decay at most r(n + 2) + d(l + p) < r(n + 3)) = R n +i local-time units 
after it was sent. Thus at the moment the node reaches threshold level R n its own 
message will already have decayed and the Counter will decrease and will be at 
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most n — 1, implying that any message sent by the node can carry a Counter of at 
most n — 1. □ 



Lemma 7.4. A stored message, (S r ,t'), that has been moved to the RUCS of a 
correct node q up to d(l + p) local-time units subsequent to the event of sending a 
message M p by p, (or was moved at an earlier time) cannot have been accounted 
for in CounterMp- 

Proof. Assume that the stored message (S r , t') was moved to the RUCS of node q 
at a local-time t, d(l+p) local-time units subsequent to the event t S endM p at node p, 
(or it was moved at an earlier time). Thus at q at local-time t, MessageAge(t, q, r) > 
r(n+l). Therefore at node p at local-time t sen dM p , MessageAge(t sen dM p ,P, r) > 
r(n + 1) — 2d(l + p) > r(n). This is because p could have received the message 
M r up to d( 1 + p) local-time units later than q did, and q could have received M p 
up to d(l + p) local-time units after it was sent. 

Following the PRUNE procedure at p, (S r , i") would have been accounted for 
at the sending time of M p only if Counter m p > n + 1. Therefore by Lemma 7.3 
node p did not account for the stored message of r in Counter m p - □ 

Corollary 7.5. A stored message, (S r ,t'), that has decayed at a correct node q 
prior to the event of sending a message M p by p, cannot have been accounted for 
in CounterM p - 

Proof. Corollary 7.5 is an immediate corollary of Lemma 7.4. □ 

Corollary 7.6. Let a correct node q receive a message M p from a correct node p at 
local-time t arr . Then, at q,from some time t in the local-time interval [t a rr,t a rr + 
d(l + p)] and at least until the end of the interval: 

\\Message_Pool\\ > CounterMp + 1 • 

Proof. Corollary 7.6 is an immediate corollary of Lemma 7.2 and Lemma 7.4. □ 

Thus, as a consequence to the lemmata, we can say informally, that when the 
system is coherent all correct nodes relate to the same set of messages sent and 
received. 



7.1 Proof of Theorem 1 

Recall the statement of Theorem 1 : 
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Any message, M p , sent by a correct node p will be assessed as timely by every 
correct node q. 

Proof. Let M p be sent by a correct node p, and received by a correct node q at 
local-time t arr . We show that the timeliness conditions hold: 
Timeliness Condition 1: < Counter m < n — 1 as implied by Lemma 7.3 and 
by the fact that the CS cannot hold a negative number of stored messages. 

Timeliness Condition 2: Following Lemma 7.3 a correct node will not fire dur- 
ing the absolute refractory period. Property 5 therefore implies that a correct node 
cannot count less than r(n + 3) local-time units between its consecutive firings. 
A previous message from a correct node will therefore be at least r(n + 2) local- 
time units old at any other correct node before it will receive an additional message 
from that same node. Following the PRUNE procedure, the former message will 
therefore have decayed at all correct nodes and therefore cannot be present in the 
Message_Pool at the arrival time of the subsequent message from the same sender. 

Timeliness Condition 3: This timeliness condition validates Counter m p - The val- 
idation criterion relies on the relation imposed at the sending node by the PRUNE 
procedure, between the M essageAge(t, p, ..) of its accounted stored messages and 
its current Counter. 

By Lemma 7.2, for all stored messages (S r ,t') accounted for in M p , 
Message Age(t,q,r) < T(Counteru p + 1) from some local-time t G [t a rr,t arr + 
d(l + p)] and until the end of the interval. 

By Corollary 7.6, \\Message_Pool\\ > Counter^ + 1, from some local-time 
t" € [tarr, tarr + d(l + p)] and until the end of the interval. 

We therefore proved that Timeliness Condition 3 holds for any < k < n at 
the latest at local-time t arr + d(l + p). 

The message M p is therefore assessed as timely by q. □ 

Lemma 7.7. Following the arrival and assessment of a timely message M p at 
node q, the subsequent execution of the MAKE- ACCOUNTABLE procedure yields 
Counter q > Counteru v - 

Proof. We first show that at time t, the time of execution of the MAKE- ACCOUNTABLE 
procedure, max[l, (Counter m p — Counter q + 1)] < ||UCS||, ensuring the exis- 
tence of a sufficient number of stored messages in UCS to be moved to CS. 

M p is assessed as timely at q, therefore, by Timeliness Condition 3 and Lemma 7.4, 
at time t, 
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Counter Mp < \\Message_Pool\\ = ||CS|| + ||UCS|| = Counter q + ||UCS|| = 

CounterM p — max[l, (CounterMp — Counter q + 1)] + 1 + ||UCS 
=>■ < — max[l, {CounterM p — Counter q + 1)] + 1 + |[UCS| 
=>■ max[l, (CounterMp — Counter q + 1)] — 1 < ||UCS|| 
=>• max[l, (CounterMp — Counter q + 1)] < ||UCS|| . 

There are two possibilities at the instant prior to the execution of the MAKE- 
ACCOUNTABLE procedure. At this instant Counter q = ||CS||: 

1. CounterMp < C ounter q , then max[l, (CounterMp — Counter q + 1)] = 
1, meaning ||CS|| will increase by 1. 

2. CounterMp > Counter q , then ||CS|| will be Counter q +max.[l, (CounterMp — 
Counter q +l)] = C ounter q +C ounter m p ~ Counter q +l = CounterM p + 

1. 

In either case, immediately subsequent to the execution of the procedure we get: 
||CS|| > CounterMp and therefore the updated Counter q > CounterMp- □ 

7.2 Proof of Lemma 3.1 

Recall the statement of Lemma 3.1: 

Following the arrival of a timely message M p , at a node q, then at time t sen d M q > 
Counter q > CounterMp- 



Proof. Let t arr denote the local-time of arrival of M p at q. Recall that t sen( iM is 
the local-time at which q is ready to assess whether to send a message consequent 
to the arrival and processing of M p . In the local-time interval [t arr , t sen dM q ] at least 
one PRUNE procedure is executed at q, the one which is triggered by the arrival of 
M p . Following Lemma 7.7, C ounter q > C 'ounter m p subsequent to the execution 
of the MAKE- ACCOUNTABLE procedure. Note that t arr < t sen d m„ < t ar r + 
d(l + p). By Lemma 7.4 all stored messages accounted for in CounterMp will not 
be moved out of the Message_Pool by any PRUNE procedure executed up to local- 
time t sen dM q , thus, Counter q must stay with a value greater than CounterMp up 
totimet sendMq - □ 
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7.3 Lemma 7.8 



Lemma 7.8. Let p,q E C% and r £ C,-, denote three correct nodes belonging to 
two different synchronized clusters. Following the arrival and assessment of p's 
and q's fires, both will be accounted/or in the Counter ofr. 

Proof. Without loss of generality, assume that p fires before node q. Following 
Lemma 4.4 node q will tire within a of p + p) on r's clock). Node r will 
receive and assess q's fire at a time i 9 at most d(l + p) + d(l +p) = 2d(l + p) after 
p fired. Summation Property [P2] ensures that r will account for each one after their 
arrival and assessments. Furthermore, MessageAge(t q ,q,p) < 2d(l + p) = r(0) 
and therefore node r did not decay or move M p to RUCS by time t q . Therefore, 
M p is still accounted for by node r at time t q and thus, both p and q are accounted 
for in Counter y at time t g . □ 
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